{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###第三周作业\n",
    "在 Rental Listing Inquiries 数据上练习 xgboost 参数调优\n",
    "数据说明：\n",
    "Rental Listing Inquiries 数据集是 Kaggle 平台上的一个分类竞赛任务，需要根据\n",
    "公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其\n",
    "中房屋的特征 x 共有 14 维，响应值 y 为用户对该公寓的感兴趣程度。评价标准\n",
    "为 logloss。\n",
    "数据链接：https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries\n",
    "为减轻大家对特征工程的入手难度，以及统一标准，数据请用课程网站提供的\n",
    "特征工程编码后的数据（RentListingInquries_FE_train.csv）或稀疏编码的形式\n",
    "（RentListingInquries_FE_train.bin）。xgboost 既可以单独调用，也可以在\n",
    "sklearn 框架下调用。大家可以随意选择。若采用 xgboost 单独调用使用方式，\n",
    "建议读取稀疏格式文件。\n",
    "关于特征工程的过程，可参看文件：FE_RentListingInqueries.ipynb\n",
    "作业要求：\n",
    "采用 xgboost 模型完成商品分类（需进行参数调优）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier # scikit-learn嵌入的\n",
    "import xgboost as xgb             #用xgboost，交叉验证做迭代次数\n",
    "\n",
    "import numpy  as np #线性代数等\n",
    "import pandas as pd #读取文件等\n",
    "import scipy  as sp #处理10^10级别的数等\n",
    "import seaborn           as sns       #画图      \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from matplotlib import pyplot\n",
    "from sklearn.model_selection import GridSearchCV  #网格搜索加交叉验证\n",
    "from sklearn.model_selection import StratifiedKFold #缺省的交叉验证用\n",
    "\n",
    "from sklearn.metrics import log_loss #评价指标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#读取数据\n",
    "trainRental = pd.read_csv(\"RentListingInquries_FE_train.csv\")\n",
    "testRental  = pd.read_csv(\"RentListingInquries_FE_test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#trainRental.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#trainRental.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#testRental.head()#观察之后发现trainRental和testRental差异在于　一维　响应值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#testRental.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sns.countplot(trainRental.interest_level)#查看响应值分布，两两之间都相差不到10倍，还可以"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#划分输入特征和响应\n",
    "X_train = trainRental.drop(\"interest_level\",axis = 1)\n",
    "y_train = trainRental[\"interest_level\"]\n",
    "#y_train.head()\n",
    "cols = X_train.columns\n",
    "#print(cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#已经是特征编码后的数据\n",
    "#StratifiedKFold 分层采样交叉切分，确保训练集，测试集中各类别样本的比例与原始数据集中相同。\n",
    "RentalKFold = StratifiedKFold(n_splits = 4,shuffle = True,random_state = 3)\n",
    "#待观察模型是过拟合还是欠拟合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]}\n"
     ]
    }
   ],
   "source": [
    "reg_alpha = [ 1.5, 2]    \n",
    "reg_lambda = [0.5, 1, 2]     \n",
    "\n",
    "param_test6 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "print(param_test6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb6 = XGBClassifier(                               #三步曲第一步\n",
    "                                                    # 在5中计算出'colsample_bytree': 0.7, 'subsample': 0.8\n",
    "                        learning_rate = 0.1,\n",
    "                        n_estimators = 383,    #首次变更项\n",
    "                        max_depth = 5,      \n",
    "                        min_child_weight = 5,   #同一考虑\n",
    "                        gamma = 0,\n",
    "                        subsample = 0.8,       #\n",
    "                        colsample_bytree = 0.7,#\n",
    "                        colsample_bylevel = 0.7,#行列采样\n",
    "                        objective = \"multi:softprob\",                                       #因为是多类分问题\n",
    "                        seed = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#GSearch_2 = GridSearchCV(xgbSecond,param_grid = param_test_2，scoring = \"neg_log_loss\",n_jobs = -1,cv = RentalKFold )\n",
    "GSearch_6 = GridSearchCV(xgb6, param_grid = param_test6, scoring='neg_log_loss',n_jobs= -1, cv=RentalKFold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=4, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=0.7,\n",
       "       colsample_bytree=0.7, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=5, min_child_weight=5, missing=None, n_estimators=383,\n",
       "       n_jobs=1, nthread=None, objective='multi:softprob', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=3, silent=True,\n",
       "       subsample=0.8),\n",
       "       fit_params=None, iid=True, n_jobs=-1,\n",
       "       param_grid={'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GSearch_6.fit(X_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58173, std: 0.00372, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.58145, std: 0.00370, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.58152, std: 0.00332, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.58229, std: 0.00389, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.58176, std: 0.00340, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.58228, std: 0.00351, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       " -0.5814514873982459)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GSearch_6.grid_scores_, GSearch_6.best_params_,     GSearch_6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([275.99703801, 269.38512474, 266.58723205, 265.26453   ,\n",
       "        290.55581522, 267.29073322]),\n",
       " 'mean_score_time': array([3.92995042, 3.93303305, 2.87339127, 2.26966006, 1.96873546,\n",
       "        1.63860059]),\n",
       " 'mean_test_score': array([-0.58173499, -0.58145149, -0.58152252, -0.58228693, -0.58176489,\n",
       "        -0.582281  ]),\n",
       " 'mean_train_score': array([-0.46838582, -0.46927708, -0.47198763, -0.46972357, -0.47083209,\n",
       "        -0.47320051]),\n",
       " 'param_reg_alpha': masked_array(data=[1.5, 1.5, 1.5, 2, 2, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_reg_lambda': masked_array(data=[0.5, 1, 2, 0.5, 1, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " 'rank_test_score': array([3, 1, 2, 6, 4, 5]),\n",
       " 'split0_test_score': array([-0.5755353 , -0.57547057, -0.57619472, -0.57588765, -0.57658622,\n",
       "        -0.57648858]),\n",
       " 'split0_train_score': array([-0.47079619, -0.47143943, -0.47505069, -0.47269024, -0.47287126,\n",
       "        -0.47594134]),\n",
       " 'split1_test_score': array([-0.58257021, -0.58226953, -0.58241465, -0.58369471, -0.58252436,\n",
       "        -0.58420067]),\n",
       " 'split1_train_score': array([-0.46760407, -0.46821981, -0.47098863, -0.46908488, -0.47026798,\n",
       "        -0.47218805]),\n",
       " 'split2_test_score': array([-0.58345733, -0.58247582, -0.58214235, -0.58320045, -0.58184326,\n",
       "        -0.5827077 ]),\n",
       " 'split2_train_score': array([-0.46821302, -0.46968931, -0.471652  , -0.46936629, -0.47011578,\n",
       "        -0.47269793]),\n",
       " 'split3_test_score': array([-0.58537794, -0.58559085, -0.58533911, -0.58636577, -0.5861065 ,\n",
       "        -0.58572779]),\n",
       " 'split3_train_score': array([-0.46692998, -0.46775977, -0.4702592 , -0.46775286, -0.47007336,\n",
       "        -0.47197474]),\n",
       " 'std_fit_time': array([0.58652365, 1.0094486 , 0.93002823, 1.95220819, 2.98501568,\n",
       "        1.1932756 ]),\n",
       " 'std_score_time': array([0.82134167, 1.31026947, 0.76146457, 0.6219798 , 0.43522309,\n",
       "        0.34540159]),\n",
       " 'std_test_score': array([0.00372066, 0.00369546, 0.00332163, 0.00388607, 0.00340044,\n",
       "        0.00351075]),\n",
       " 'std_train_score': array([0.00146376, 0.00143751, 0.00183579, 0.00181799, 0.00117953,\n",
       "        0.00160408])}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GSearch_6.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.581451 using {'reg_alpha': 1.5, 'reg_lambda': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Users\\Dell\\Documents\\anaconda\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd809Uax/HPQzd7DykICsgWoWAZoiAiQwQRFJwgwwEqeN3rclGvoFwHgjJEwAUqDkBABGULQtlbhowKMgoCpdB57h8nxVJbmo70l6TP+/XqiyT9JXlSmn5zxu8cMcaglFJK5bVCTheglFLKP2nAKKWU8ggNGKWUUh6hAaOUUsojNGCUUkp5hAaMUkopj9CAUUop5REaMEoppTxCA0YppZRHBDpdgJPKli1rqlWr5nQZSinlU9auXXvcGFMuq+MKdMBUq1aNqKgop8tQSimfIiL73TlOu8iUUkp5hAaMUkopj9CAUUop5REaMEoppTxCA0YppZRHaMAopZTyCA0YpZRSHqEBkxNn/oSV7ztdhVJKeTUNmJxYOxXmPwdRHzldiVJKZV/sUUhJ8fjTaMDkROsnoUY7mPsU7FvhdDVKKeWe5ERYORZGN4b1n3j86TRgcqJQANw+CUpVhy/vhZNurZqglFLO2bcCxreG+c9D1Uio1srjT6kBk1NhJaH3dEhOgul3QXys0xUppdQ/nTkC3wyEKZ3s36len8PdX0GZKz3+1BowuVG2BvScDEe3wbcP5kufplJKuSU5CVZ9AGMiYOu3cN2TMOhXqN0ZRPKlBA2Y3KpxI7R/DXZ8D0tGOF2NUkrB/pUw4Xr44VkIbwqPrIIbX4LgwvlaRoFerj/PRD4MR7bCkpFQvg7Uu83pipRSBVHsUVjwb9j4ORQPhzs+gTpd8q3Fkp4GTF4QgVveguO/wbcPQ+kroNLVTlellCookpPsaRM/vwqJcdDqCTvbNbiIo2VpF1leCQyBOz+FwqVh2l32k4RSSnnagV9h4g0w7ymo3BgeWQnt/u14uIAGTN4qVsHO0IiLgS/ugaR4pytSSvmr2GPw3SD4qD3EnYCeU+Heb6FsTacru0ADJq9d1gi6vQ8Hf4U5T4AxTleklPInKcmweiKMaQKbpkPLITBoNdTr5thYS2Z0DMYT6ne3U5eXvgkV6ttJAEoplVsH18Dcf8HhjVC9NXQaBeWucrqqTGnAeMoNz8PR7fas2bK17HRmpZTKibPHYeEwu7xLsUrQY7KdreplLZb0tIvMUwoVgtvGQ7k6MKMvxOxxuiKllK9JSYY1k+C9JrBxGrR4DAavsb0kXh4uoAHjWSFFoffnIAEwrRecP+V0RUopXxG9Fj680Y7lVmwAD62A9q9ASDGnK3ObBoynlaoGd34CJ/bCjH72E4lSSmUm7gTMesyGy+nDdmHd+2dD+dpOV5ZtGjD5oVor6PQm7F5g+1GVUiq9lBRYOwXeawzrP4Xmg2x3WIMePtEdlhGPBoyIdBCRnSKyW0SezeD7fUTkmIhscH31T/O9N0Rkq4hsF5HRYhUWkTkissP1vRFpjn9CRLaJyCYR+UlELvfka8u2iAegaX/4ZTRsmOZ0NUopb/LHOpjUDmY/DuXrwkPL4ebXILS405XlisdmkYlIADAWuAmIBtaIyCxjzLZ0h35hjBmc7r4tgJZAQ9dNy4HrgdXAKGPMIhEJBn4SkY7GmHnAeiDCGBMnIg8DbwB3eur15UiHEXBsJ8x+DMrUgCpNna5IKeWkuBPw8ysQNRmKlofuE6FBT59tsaTnyRZMM2C3MWavMSYBmA50dfO+BggFgoEQIAg4YoyJM8YsAnA95jog3HV9kTEmznX/Vam3e5WAIHu2bbFK8MXdcPqQ0xUppZyQkgLrPrazw9ZOtefKDV4DDe/wm3ABzwZMZeBgmuvRrtvSu93VrTVDRKoAGGNWAouAw66v+caY7WnvJCIlgS7ATxk8Zj9gXu5fggcUKWM3Kks4azcqSzzndEVKqfx0aANMuglmPWpPknxwKXR4HUJLOF1ZnvNkwGQUw+nXTZkNVDPGNAQWAlMBRKQGUAfbCqkMtBWR1hceWCQQmAaMNsbsvehJRe4BIoA3MyxKZKCIRIlI1LFjx3L0wnKtQl3bFD60AWYO1uVklCoIzp2EOf+CCTfAX/vteXJ950HF+k5X5jGeDJhooEqa6+HARX1CxpgYY0zqipATgSauy7cBq4wxscaYWGxrJDLNXScAu4wx76R9PBFpB7wA3JrmcS9ijJlgjIkwxkSUK1cuhy8tD9TuBG1fhC0zYPnbztWhlPKslBQ7K+y9CLuk/rUPwuAouLqXX3WHZcSTAbMGqCki1V0D8r2AWWkPEJFKaa7eCqR2gx0ArheRQBEJwg7wb3fd51WgBDAk3WNdA4zHhotvrJV/3b+g/u3w03DY6Z09ekqpXDi8CT66GWYOgjJXwsAl0HEkhJV0urJ84bFZZMaYJBEZDMwHAoCPjDFbRWQ4EGWMmQU8JiK3AknACaCP6+4zgLbAZmy32g/GmNkiEo5toewA1olN/zHGmA+xXWJFga9ctx8wxtzqqdeXJ0Tg1jF2GZmv+0P/hXZHTKWUbzv3Fyz6L6yZCGGloev7cHVvu4RUASKmAPf/R0REmKioKKfLgFN/2H7Z4MIwYJHdtEwp5XuMgY3TYcFLdl+oiH7Q9gUIK+V0ZXlKRNYaYyKyOq5gxam3KlHZblR2+hB8eR8kJzpdkVIqu/7cApM7wncP2SWiBiyCzqP8LlyyQwPGW1RpCl3ehX3L7BL/SinfcP4UzHsWxreG47/Zbu8HfrSbDxZwuh+MN2l0FxzZCivH2OUiIvo6XZFSKjPGwKYv4ccX4ewxuxxU2xe1izsNDRhvc9NwOLYD5j5pNyqr1tLpipRS6R3ZCnOehAO/QOUmcNcXULmx01V5He0i8zaFAuzy3KWqw5f3wsn9TleklEp1/jT88DyMu85+EOwyGvot1HDJhAaMNworaZeTSU6yy8nExzpdkVIFmzGw6SsYEwGr3ofG98Kja6HJ/QVu6nF26E/GW5WtAT0/gqPb4NsH7dnASqn8d3Q7TO0C3/SH4pdB/5/shBwda8mSBow3q9EO2r8KO76HJSOyPl4plXfiz9gB/HGt4M/NcMvbNlzCm2R9XwXoIL/3i3zEDiguGWnP8q93m9MVKeXfjIGt38D8F+DMYWh8H9w4zK6ErrJFA8bbidhPTsd3wbcPQ+kroNLVTlellH86ttPO4Px9qX2f3fkphGd5wrrKhHaR+YLAEPuLXrg0TLsLYn1jLU+lfEZ8LCx4GT5oAYc3Quf/2TPxNVxyRQPGVxSrYJeTiYuBL+6BpAx3I1BKZYcxsPVbGNMUVrxrl9B/dB007W9PGVC5ogHjSy5rBN3GwsFfYc4TulGZUrlx7Df4pBt81ceOr/RbAF3HQpGyTlfmN3QMxtfUv91Om1z6JlRoAJEPOV2RUr4l4ax9//wyBoIKQ6dRdpkXbbHkOQ0YX3TD83BkG8x/DsrVgivbOl2RUt7PGNg+y56JfzoaGt0N7YZB0fJOV+a3tIvMFxUqBN3HQ7natnkfs8fpipTybsd3w6e32+0wwkrCA/Oh2/saLh6mAeOrQopB72kgATCtl10yXCl1sYQ4uyX5B80heg10GGm3La4a6XRlBYIGjC8rVQ3u+BhO7IUZ/SAl2emKlPIOxsD22TC2GSz7H9TrDoOj7JhlgI4M5BcNGF9X/Tro+AbsXgALhzldjVLOi9kDn/W00/lDikGfubZLuVgFpysrcDTK/UHTfnY5mV9GQ4V6di6/UgVNQhwsfxtWvAMBIXDz69BsAAQEOV1ZgaUB4y86jrTbtc56DMrU0DOQVcFhDOycBz88A38dgAZ3QPtXoFhFpysr8LSLzF8EBEHPqfZNNf1uOH3I6YqU8rwTe+HzO2F6bwgqAn3mwO0TNVy8hAaMPylSxm5UlhBrNypLPOd0RUp5RuI5WPQ6jI2E/Sug/Wvw0DKo1srpylQaGjD+pkJd6D4BDm2AWY/qcjLK/+z8AcZea/dIqtPFzg5rMVjHWryQBow/qt0Z2r4Im7+yA55K+YOT++DzXjDtTggMhftmQY9JULyS05WpTOggv7+67l92ZtnC/9gz/q/q6HRFSuVM4nm70vHyt+yJxTcNh2sfhsBgpytTWdCA8VcidmXYE3vh6/7Qf6HdEVMpX7JrAcx9Ck7+bndzbf8alKjsdFXKTdpF5s+CC9s9ZIIK2+Vk4k44XZFS7jm5386G/KwHFAqEe7+DnlM0XHyMBoy/K1EZen1mpy1/dT8kJzpdkVKZS4q3S+mPvRb2/GxXO374F7iyjdOVqRzQgCkIqjSDLu/afcbnP+90NUplbPdCeD8Sfn4VarWHwWug1VAda/FhOgZTUDS6yw76rxwD5etCRF+nK1LK+uug3dto+2y7CsU930CNG52uSuUBDZiC5KbhcGwHzH0SytaCai2drkgVZEnx9gPP0lH2fK0bX4bmgyEwxOnKVB7RLrKCpFAA3D7JLvP/5b12IFUpJ+z5GT5oYfdqubItDF5tp9ZruPgVDZiCJqykXU4mOckuJxMf63RFqiA59Qd8eT98chuYFLh7hp2EUrKq05UpD9CAKYjK1oSeH8HRbfDtg5CS4nRFyt8lJcDyd2BMU/jtB2jzIjy8Emre5HRlyoM0YAqqGu2g/auw43tYMtLpapQ/27sYxrWEhf+GK26AQavh+qcgKNThwpSn6SB/QRb5iJ1ZtmQElK9tz5RWKq+cPgTzX4Ct39hxv7u+hFo3O12VykcaMAWZCNzyNhzfBd8+DKWvgEpXO12V8nXJibDqA1g8Akwy3PA8tHxcWywFkEe7yESkg4jsFJHdIvJsBt/vIyLHRGSD66t/mu+9ISJbRWS7iIwWq7CIzBGRHa7vjUhzfIiIfOF6rl9FpJonX5vfCAyBOz+FwqVh2l0Qe9TpipQv+30pjGsFC16C6q3hkVVwwzMaLgWUxwJGRAKAsUBHoC7QW0TqZnDoF8aYRq6vD133bQG0BBoC9YGmwPWu40cZY2oD1wAtRSR1meB+wEljTA3gbUAHFtxVrIJdsywuBr64156foFR2nD4MM/rB1C52M7De0+Gu6VC6utOVKQd5sgXTDNhtjNlrjEkApgNd3byvAUKBYCAECAKOGGPijDGLAFyPuQ4Id92nKzDVdXkGcKOISJ68koLgskbQbSwcXAVzntCNypR7khPhlzEwJsKeiX/9MzDoV90eQgGeHYOpDBxMcz0auDaD424XkdbAb8BQY8xBY8xKEVkEHAYEGGOM2Z72TiJSEugCvJv++YwxSSJyCigDHM/D1+Tf6t8OR7bBslFQoQFEPuR0Rcqb7VthV4U4ug1qtoeOI+04nlIunmzBZNR6SP+xeDZQzRjTEFiIqwUiIjWAOtjWSWWgrSuEcH0/EJgGjDbG7M3G8yEiA0UkSkSijh07ls2XVAC0eQGu6mzXhtrzs9PVKG905k/4ZiBM6WRP1O31uZ0hpuGi0vFkwEQDVdJcDwcOpT3AGBNjjEnt8J8INHFdvg1YZYyJNcbEAvOAyDR3nQDsMsak3Q/4wvO5AqgE8I8NUIwxE4wxEcaYiHLlyuX4xfmtQoWg+3i7C+ZXfSBmj9MVKW+RnGRnh41pClu/hdZP2e6w2p3tjESl0vFkwKwBaopIdREJBnoBs9IeICJpN9O+FUjtBjsAXC8igSIShB3g3+66z6vY8BiS7vlmAfe7LvcAfjZGBxJyJKQY9J5mt6ed1gvOn3K6IuW0/b/A+Nbww7MQ3tTODmv7ot3UTqlMeCxgjDFJwGBgPjYcvjTGbBWR4SJyq+uwx1zTjTcCjwF9XLfPAPYAm4GNwEZjzGwRCQdewM5KW5duavMkoIyI7AaeAP4xLVplQ6lqcMfHf2+5nJLsdEXKCbFH4duHYHJHiD9tp7Tf8zWUudLpypQPkIL8IT8iIsJERUU5XYZ3WzPJzipr+bhd7l8VDMlJEDXJbv6VeA5aPmZXOw4u4nRlyguIyFpjTERWx+mZ/OrSmvazy8mseNduVHZ1L6crUp52YBXMeRKObIYr2kCnUVC2htNVKR+kAaOy1nEkHP8NZj1mdxwMz/KDi/JFscfsgpQbPoPilW0XaZ1bdQBf5ZiupqyyFhAEPadCsYow/W67iKHyHynJsHoijGkCm76EVkPtisd1u2q4qFzRgFHuKVLGLv+REGtDJvGc0xWpvHBwNUy4wZ4wWakRPPwLtBsGIUUdLkz5g2wFjIgUEpHinipGebkKdaH7BDi0DmY9qsvJ+LKzx2HmIJh0k73cYzLcNxPK1XK6MuVHsgwYEflcRIqLSBFgG7BTRJ7yfGnKK9XubM9/2PwVrHgn6+OVd0lJhjUfwntNYON0aPEYDF4D9btrd5jKc+4M8tc1xpwWkbuBucAzwFrgTY9WprzXdU/aNcsW/gfK1YGrOjhdkXJH9Fo75fzwBqh2nZ0dVr6201UpP+ZOF1mQ62z6bsBMY0wiGazxpQoQEeg6Fio1tCdhHt2e9X2Uc87G2BmAH95o1xG7fRLcP1vDRXmcOwEzHtgHFAGWisjlwGlPFqV8QHBhu8hhUJhdTibuH8u+KaelJEPUZDs7bP2n0HwQPBoFDXpod5jKFzk6k19EAl1Lwfg0PZM/DxxcDVM6Q9VIuOcbO6VZOe+PtfZkyUPr4PJW0OlNO0nDyyQmJhIdHc358+edLkVlIDQ0lPDwcIKCLn5f59mZ/CLyODAZOAN8iN1J8lngxxxVrPxLlWbQ5V347mGY/wJ0esPpigq2uBPw03BYOwWKlofuH3p1iyU6OppixYpRrVo1dH9A72KMISYmhujoaKpXz9nOpO50kT1gjDkNtAfKAX2BETl6NuWfGt0FzQfD6vH2D5vKfykpsHaqnR227mOIfAQGR0HDnl4bLgDnz5+nTJkyGi5eSEQoU6ZMrlqX7swiS/2f7wRMNsZs1K2I1T+0+48d7J/zLyhTE6q1dLqiguPQetsd9kcUVG0BnUdBhXpOV+U2/XPivXL7f+NOC2atiPyIDZj5IlIMSMnVsyr/ExAIPT6yy/x/eS/8dcDpivxf3An4/gmY0Mb+vG+bAH3n+lS4KP/mTsD0w465NDXGxAHB2G4ypS4WVtIuJ5OcBNN62+10Vd5LSYF1n8CYCFg7Ga590J4sefWdXt0dpmDx4sXccsstuT7mUnbs2EHz5s0JCQlh1KhRmR7Xp08fqlevTqNGjWjUqBEbNmzI8XNmJssuMmNMimujr7tczaUlxpjZeV6J8g9la0LPj+CznvDdQ9DzY7sNs8obhzfa7rDo1VAl0naHVWzgdFV+wxiDMYZCPvw7W7p0aUaPHs13332X5bFvvvkmPXr08Fgt7swiGwE0BT5z3fSYiLQwxjznsaqUb6vRDm56BX58AZaMhDb6q5Jr5/6CRa/ZZV7CSkO3D6BhL78K7//M3sq2Q3l7il3dy4rz7y6X7jLct28fHTt2pE2bNqxcuZIhQ4Ywbtw44uPjufLKK5k8eTJFixZl7ty5PPHEE5QtW5bGjRuzd+9evv/++wwfc/Xq1QwZMoRz584RFhbG5MmTueqqqy46ZtiwYezZs4c//viDgwcP8vTTTzNgwAAAYmNj6dGjB1u2bKFJkyZ8+umniAjDhw9n9uzZnDt3jhYtWjB+/Ph/jJOUL1+e8uXLM2fOnFz85PKGO7+dnYCbjDEfGWM+AjoAnT1blvJ5zQdBo7thyQjYmvUnKZWJlBTY8LmdHbbmQ2jaHx5da2fu+VG4OG3nzp3cd999LFiwgEmTJrFw4ULWrVtHREQEb731FufPn+fBBx9k3rx5LF++nGPHjl3y8WrXrs3SpUtZv349w4cP5/nnn8/wuE2bNjFnzhxWrlzJ8OHDOXTIboWxfv163nnnHbZt28bevXtZsWIFAIMHD2bNmjVs2bKFc+fOXQi4cePGMW7cuGy/7hdeeIGGDRsydOhQ4uPjs33/rLi74VhJIPVU7RJ5XoXyPyJwy9twfJc9R6b0FXZpGeW+Pzfb7rCDqyC8GXT+Bipd7XRVHpNVS8OTLr/8ciIjI/n+++/Ztm0bLVvaWZAJCQk0b96cHTt2cMUVV1w4H6R3795MmDAh08c7deoU999/P7t27UJESExMzPC4rl27EhYWRlhYGG3atGH16tWULFmSZs2aER4eDkCjRo3Yt28frVq1YtGiRbzxxhvExcVx4sQJ6tWrR5cuXXjooYey/Zpff/11KlasSEJCAgMHDmTkyJG8/PLL2X6cS3HnI9DrwHoRmSIiU7ELXf43T6tQ/ikwBO78FMJK2UH/2KNOV+Qbzp+Cec/A+NYQs8uu+/bAfL8OF6cVKVIEsGMwN910Exs2bGDDhg1s27aNSZMmkd0VT1566SXatGnDli1bmD17dqbnkqTv3kq9HhIScuG2gIAAkpKSOH/+PI888ggzZsxg8+bNDBgwIFfnqFSqVAkRISQkhL59+7J69eocP1ZmsgwYY8w0IBL4xvXVHFia55Uo/1SsAvT6DOJi4It7ISnB6Yq8lzF2Cf33IuDX8RDxgD1Z8pp7tDssn0RGRrJixQp2794NQFxcHL/99hu1a9dm79697Nu3D4Avvvjiko9z6tQpKleuDMCUKVMyPW7mzJmcP3+emJgYFi9eTNOmTTM9NjVMypYtS2xsLDNmzMjGK/unw4cPAzZUv/vuO+rXr5+rx8uIW7+1xpjDxphZxpiZxpg/gVV5XonyX5ddA93G2q6eOU/oRmUZObIVJneCbx+EklVh4CLo/D8oXNrpygqUcuXKMWXKFHr37k3Dhg2JjIxkx44dhIWF8f7779OhQwdatWpFhQoVKFEi89GCp59+mueee46WLVuSnJyc6XHNmjWjc+fOREZG8tJLL3HZZZdlemzJkiUZMGAADRo0oFu3bheFUdoxmD///JPw8HDeeustXn31VcLDwzl92k6e6NSp04VxnrvvvpsGDRrQoEEDjh8/zosvvpitn5U7crrY5UFjTJU8ryaf6WKX+eynV2DZKOgwEiKz32fsl86fhsWv2xZLaAm7XfE19xaYFsv27dupU6eO02W4JTY2lqJFi2KMYdCgQdSsWZOhQ4fm+PGGDRtG0aJFefLJJ/OwyryX0f9Rni12mQn9CKqyr80LdjmZ+c/ZrXmvbOt0Rc4xBjbPsFO5Y49Ckz5w48vaYvFiEydOZOrUqSQkJHDNNdfw4IMPOl2S18u0BSMi75FxkAhwvzGmuCcLyw/agnFA/BmY1B5OH4IBP0OZK52uKP8d3W5nh+1fbrsPO/8PKjdxuipH+FILJiOTJ0/m3Xffvei2li1bMnbsWIcqynueasFc6i+v/lVWORNSDHpPs+tnTesN/RfYrqGCIP4MLB4Bqz6A0OJwyzvQ+D4oFOB0ZSqH+vbtS9++unJWZjINGGPM1PwsRBUgparBHR/DJ93slsu9p/v3H1ljYMvX8OOLdsvixvfBjf+GImWcrkwpjyoYI4nK+1S/DjqOhF0/wk//cboazzm6A6Z2ga/7QdEK0H8h3Dpaw0UVCDkd5Fcq95r2t9NzV7wL5evZ1YD9RXysXYdt1fsQXBQ6v2UH8v25paZUOhowylkd37DLycx61A74h2c5bujdjIGt39rto88cslOO2w2DImWdrkypfJdlF5mIjM7g6xUR6ZofBSo/FxAEPafaM/6n321nl/mqY7/Bx11hRl8bKP0WQNcxGi7qgvzYD+azzz6jYcOGNGzYkBYtWrBx48YcP1ZuuTMGEwo0Ana5vhoCpYF+IvKOB2tTBUWRMnagPyHWhkziOacryp6Es7Dg3/BBCzi8ATqNgoGLoUozpytT2WSMISXFtzfsrV69OkuWLGHTpk289NJLDBw40LFa3OkiqwG0NcYkAYjIB8CPwE3AZg/WpgqSCvWg+wSYfhfMesxe9vbdGY2BbTNtd9jpaLs9Qbv/QNFyTlfmm+Y9a1eQzksVG0DHEZc8xN/2g2nRosWFy5GRkURHR+fkJ5cn3GnBVAaKpLleBLjMGJMM5P0GAqrgqt0Z2r4Im7+EFV7eOD6+Gz7tDl/db1eLfmA+dHtfw8VH+et+MJMmTaJjx465+dHkijstmDeADSKyGHsWf2vgvyJSBFjowdpUQXTdk3BkGyz8D5SrA1d1cLqiiyWchWX/gxWjISjMTlKI6AcBOl8m17JoaXiSP+4Hs2jRIiZNmsTy5ctz++PJsSzfFcaYSSIyF2iGDZjnjTGpI7FPebI4VQCJ2P1PTuyxJ2H2Xwjlaztdle0O2/E9/PAcnDoIV/eGm4ZD0fJOV6byQPr9YKZNm3bR99evX5+tx0vdD+bbb79l37593HDDDRkel5P9YKKioqhSpQrDhg3LdD+YTZs20b9/f+bNm0eZMs6dc+XuiZZNgeuAVkDBXDRJ5Z/gwtDrc9tCmNYL4k5kfR9PitkDn/WAL+6BkOLQdx7cNk7DxQ/5w34wBw4coHv37nzyySfUqlXrknV6mjvTlEcAjwPbXF+Picjrni5MFXAlwu1GZaf/gK/6QHLGXQwelRAHP78K70fCgV+hwwh4cClc3iLr+yqf5A/7wQwfPpyYmBgeeeQRGjVqRESEc+eWZbkfjIhsAhoZY1Jc1wOA9cYYn99gXVdT9gHrP4WZg6DZg9Dpjfx5TmNg51w7q+nUAWh4p+0OK1Yxf56/APGl1ZR1P5i/ubuasrtdZCXTXHZ76VsR6SAiO0Vkt4g8m8H3+4jIMRHZ4Prqn+Z7b4jIVhHZ7jq5U1y3vyYiB0UkNt1jVRWRRSKyXkQ2iUgnd+tUXuyaeyByEKweD2uneP75TuyFz++w06WDi0CfOXbKtIZLgTdx4kQaNWpEvXr1OHXqlO4H4wZ3WjC9gRHAIv6eRfacMWZ6FvcLAH7Dni8TDawBehtjtqU5pg8QYYwZnO6+LYA3Xc8FsNz1nItFJBLYD+wyxhRNc58J2JbVByJSF5hrjKl2qRq1BeMjkpPsH/3fl8L9szzTRZV4Dpa/DcvfsasLtHkemg20l5XH+FILJiO6H8yluTOLbJprinJTbMA8g3stn2bAbmPMXldB04Gu2HGcLJ8Wu4JAsOs5g4AjrnoM7NBNAAAYtElEQVRWuR4vo/ukboJWAvDhNUfURQICocdH8OGNdqB94GK7b31e2fkDzHsa/toP9XtA+1eheKW8e3zlt3Q/mEtzq4vMGHPYGDPLGDPTGPMnsMqNu1UGDqa5Hu26Lb3bXV1aM0Skiuv5VmJbTIddX/ONMduzeL5hwD0iEg3MBR51o0blK8JK2uVkkpPsRmXxsVnfJysn98HnvWDanRAYCvfPhh6TNFzyWVa9KMo5uf2/yel+MO6s4ZHRMemrnQ1Uc00YWAhMBRCRGkAdIBwbSm1FpDWX1huYYowJBzoBn4jIP16fiAwUkSgRicrqbFzlZcrWhJ4fwdFt8N3DkNM1oxLPw+KRMPZa2+120yvw8AqontWvmMproaGhxMTEaMh4IWMMMTExhIaG5vgxcnr6sTu/DdFAlTTXw0nXbWWMiUlzdSIw0nX5NmCVMSYWQETmAZHA0ks8Xz+gg+txV4pIKFAWOJruOScAE8COwbjxOpQ3qdHOBsKPL9j9Vto8l737//YjzHvKtl7qdbfdYSUyalir/BAeHk50dHSWS68oZ4SGhl5YUSAnMg0YEXmPjINEuHhWWWbWADVFpDrwB9ALuCvdc1Qyxhx2Xb0VSO0GOwAMcJ1vI8D1QFaLUx0AbgSmiEgd7BiO/tb6o+aD7EZlS0ZA+TpQr1vW9zm5356Fv3MOlK0F982EK27wdKUqC0FBQReWX1H+51ItmEtNr8py6pUxJklEBgPzgQDgI2PMVhEZDkQZY2ZhT9q8FUgCTgB9XHefAbTFrtZsgB+MMbPBTl/GBlVh13jLh8aYYcC/gIkiMtR1nz5G293+SQRueRtidtmustJXQKVMTstKPA+/vAfLRoEUsqsdRz4CgcH5W7NSBVCW05QvOlikomuQ3y/oNGUfd+YITGxjg2PAon+uZLxroe0OO7EX6naDm1+zKwQopXIlr0+0TDU3h/UolfeKVbDLyZw9Bl/eC0kJ9va/DtiNyz673YbPvd/CHVM1XJTKZ9kd5PfyHaBUgXPZNXb15a/7wZyhUKo6LB1lv3fjy9B8MASGXPoxlFIekd2AmeiRKpTKjQY97NTlZf+z1+t0gZtfh5JVLn0/pZRHZStgjDHve6oQpXKlzYt2Kf2K9e1UZqWU43QbPuUfChWCVkOcrkIplUZOz+RXSimlLkkDRimllEdowCillPIIDZgcSkrO4UKLSilVQGjA5MDczYe55b3lbPnjlNOlKKWU19KAyYGw4ABizibQdewK/vfjTuKTkp0uSSmlvI4GTA60uao8C4a2pmujy3jv5910eW85m6L/crospZTyKhowOVSycDBv3dGIj/pEcPpcEt3GrmDEvB2cT9TWjFJKgQZMrrWtXYEfn2hNzyZVGLdkD51HL2PdgZNOl6WUUo7TgMkDxUODGNmjIVMfaMa5hGR6fPALr83Zpq0ZpVSBpgGTh66vVY75Q1vTq1lVJi77nU7vLiNq3wmny1JKKUdowOSxYqFB/Pe2BnzW/1oSklPoOX4l/5m9lbiEJKdLU0qpfKUB4yEta5Rl/pDW3Bt5OZNX7KPju8tYtTfG6bKUUirfaMB4UJGQQIZ3rc/0gZEA9JqwipdnbuFsvLZmlFL+TwMmH0ReUYZ5j1/HAy2r88mq/dz8zlJW7D7udFlKKeVRGjD5pHBwIC93qctXDzYnKKAQd3/4K89/u5kz5xOdLk0ppTxCAyafRVQrzbzHr2Ng6yuYvvoAN7+9lKW/HXO6LKWUynMaMA4IDQrg+U51mPFwC8KCA7jvo9U8M2MTp7U1o5TyIxowDmpctRRzHruOh2+4kq/WHqT9W0tZtOOo02UppVSe0IBxWGhQAM90qM13g1pSIiyIvlPW8MSXGzgVp60ZpZRv04DxEg3DSzLr0ZY82rYGMzccot3bS1iw7YjTZSmlVI5pwHiRkMAA/tX+KmYOaknZoiEM+DiKx6ev5+TZBKdLU0qpbNOA8UL1K5dg5qCWDGlXkzmbDnPT20v4Ycthp8tSSqls0YDxUsGBhRjSrhazBreiQvFQHvp0HYM+X0dMbLzTpSmllFs0YLxc3cuK892gljzZvhYLth7hpreXMnvjIYwxTpemlFKXpAHjA4ICCjG4bU2+f6wVVUqF8ei09Tz86TqOndHWjFLKe2nA+JBaFYrx9cMteLZjbX7eeZSb3l7Cd+v/0NaMUsoracD4mMCAQjx0/ZXMfew6qpctwpAvNjDg47UcPX3e6dKUUuoiGjA+qkb5osx4qAUvdq7Dsl3HaPfWEmasjdbWjFLKa2jA+LCAQkL/665g3uPXcVXFYjz51UYemLKGw6fOOV2aUkppwPiDK8oV5YuBzfl3l7qs2nuC9m8t5Ys1B7Q1o5RylAaMnyhUSOjbsjo/DLmOepWL88zXm7nvo9X88Ze2ZpRSztCA8TOXlynC5/0jeaVrPdbuP0n7t5bw2a/7tTWjlMp3Hg0YEekgIjtFZLeIPJvB9/uIyDER2eD66p/me2+IyFYR2S4io0VEXLe/JiIHRSQ2g8e7Q0S2ue73uSdfmzcrVEi4t3k15g9pTaOqJXnh2y3c/eGvHDwR53RpSqkCxGMBIyIBwFigI1AX6C0idTM49AtjTCPX14eu+7YAWgINgfpAU+B61/GzgWYZPF9N4DmgpTGmHjAkj1+Sz6lSujCf9ruW/97WgE3Rp7j5naV8vHIfKSnamlFKeZ4nWzDNgN3GmL3GmARgOtDVzfsaIBQIBkKAIOAIgDFmlTEmo5UfBwBjjTEnXcfpzl2AiHDXtVWZP7Q1EdVK8/LMrfSeuIp9x886XZpSys95MmAqAwfTXI923Zbe7SKySURmiEgVAGPMSmARcNj1Nd8Ysz2L56sF1BKRFSKySkQ65P4l+I/KJcOY2rcpb/RoyLbDp+nw7lImLf+dZG3NKKU8xJMBIxnclv6v2WygmjGmIbAQmAogIjWAOkA4NpTaikjrLJ4vEKgJ3AD0Bj4UkZL/KEpkoIhEiUjUsWPHsvFyfJ+IcEdEFRYMvZ4WV5blle+3ccf4lew59o/hLKWUyjVPBkw0UCXN9XDgUNoDjDExxpjUFRsnAk1cl28DVhljYo0xscA8INKN55tpjEk0xvwO7MQGzkWMMROMMRHGmIhy5cpl+0X5g4olQpl0fwRv3XE1u4/G0undZUxYukdbM0qpPOXJgFkD1BSR6iISDPQCZqU9QEQqpbl6K5DaDXYAuF5EAkUkCDvAn1UX2XdAG9fjlsV2me3N9avwUyJC98bhLBjamta1yvHfuTu4/YNf2H30jNOlKaX8hMcCxhiTBAwG5mPD4UtjzFYRGS4it7oOe8w1pXgj8BjQx3X7DGAPsBnYCGw0xsyGC9OXo4HCIhItIsNc95kPxIjINuz4zVPGmBhPvT5/Ub54KBPubcK7vRqxP+YsnUYv5/3Fu0lKTnG6NKWUj5OCfAJeRESEiYqKcroMr3HsTDwvz9zCvC1/0jC8BG/2uJqrKhZzuiyllJcRkbXGmIisjtMz+dUF5YqF8ME9TRh7V2P+OHmOW95bxuifdpGorRmlVA5owKh/6NywEj8ObU2H+pV4a8FvdBu7gm2HTjtdllLKx2jAqAyVKRrCe72vYdw9TThyOp5bxyzn7QW/kZCkrRmllHs0YNQldahfkQVDW9Pl6st496dd3DpmOVv+OOV0WUopH6ABo7JUqkgwb9/ZiA/vi+BkXAJdx65g1PydxCclO12aUsqLacAot7WrW4Efh1zPbddUZsyi3XR5bzkbDv7ldFlKKS+lAaOypUThIEb1vJrJfZty5nwS3d9fwevztnM+UVszSqmLacCoHGlzVXnmD23NHRFVGL9kL51GL2Pt/pNOl6WU8iIaMCrHiocGMeL2hnzSrxnxiSn0GPcLr36/jXMJ2ppRSmnAqDxwXc1yzB/amruaVeXD5b/T8d2lrP79hNNlKaUcpgGj8kTRkEBeu60Bn/e/lmRjuHPCSobN2kpcQpLTpSmlHKIBo/JUixpl+eHx1tzfvBpTftlHh3eWsXKPrjmqVEGki13qYpce8+veGJ7+ehP7Y+K4J7Iqz3asQ9GQQKfLUqrAiEtIIiY2gRNn7dfx2PgLl2+uX5HGVUvl6HHdXexS3+3KY669ogw/PN6aUT/u5KMVv7NoxzFG3t6QVjXLOl2aUj7HGENcQjIxsQnEnLVBEeMKi5jY+AuX7XV7zPnEjJd2CgksRPWyRXIcMO7SFoy2YPLF2v0neOqrTew9fpbezarwfKc6FAsNcrospRxjjCE2PulCUNiWhisoXK2O42ftbSdi7THxmawFGBpUiDJFQihTNJjSRexXmSLBlHbdVubCbfZ64eAARDLa1d497rZgNGA0YPLN+cRk3l7wGxOX7aVi8VBev70h19cqmNtWK/9jjOFMfNLfQeEKiX+ER5pWRkImW2EUDg5IExLBlCkacuGyvW7DIvVy4eD87YzSgHGDBowz1h84ydMzNrHraCw9m4Tz4i11KRGmrRnlXYwxnD6XdKE76viFsYz4NF1Tqd1U9pjE5Iz/nhYJDqC0KxQuBIWrZVGmSMiFy6mtjLDggHx+tdmjAeMGDRjnxCclM/qnXYxbspeyRYP5720NuLFOBafLUn4sJcVw6lximlZEfJrWxd9BkRoaJ88mkJSS8d/HYiGBlC76d1dU+pAoXSSYskVDLlwODfLuwMguDRg3aMA4b3P0KZ78aiM7j5yh+zWVeblLXUoWDna6LOUDUlIMf51L5MTZ+Auti7/HL+Lt+EWa20/GJZCcWWCEBtqgcIXCpbqmShcJJiTQvwIjuzRg3KAB4x0SklIYs2g37y/aTakiwbzarT4316vodFkqnyWnGE7G/d31lNrKOJ5mmu2F2VOxNjAyyQtKhAWlC4aQNKFx8YB3qcLBBAfqKYHZoQHjBg0Y77L10Cme/GoT2w+f5tarL2PYrfUoXURbM74qKTmFk3GJ/5hGm74rKjU8TsYlkNmfo5KFg2y3k2tg+6Ixi7ThUSSYUkWCCQrQwPAkDRg3aMB4n8TkFD5YvIf3ft5FibAghnetT6cGlZwuS2H/b06mPffCFRwXd00lcNzVyjh1LjHDwBCBUoUvnk5rWxUXB0Vqd1WpwkEEamB4FQ0YN2jAeK8df57mqa82sfmPU3RqUJHhXetTtmiI02X5lYSkFE7GJVx84l7s311RMRd1TdnAyEghV2Ck7Xr6e6DbBkfarqlShYMJKJTzczCU8zRg3KAB492SklMYv3Qv7y7cRZGQAP7TtT5dGlbK1Qli/iw+KTnd+MXFS4OkP+v7zPmMFyINKCQ2MNJMpy3rOmkvbddUaniUCAvSwChgNGDcoAHjG3YdOcOTMzax8eBftK9bgVdvq0/5YqFOl+Vx5xP/DozUFkaGXVOu7qkz8RkHRmAhoVRmXVEXAuPv2VMlwoIopIGhLkEDxg0aML4jKTmFSct/538LfiMsKIBht9alW6PKPtWaOZeQfPEaUq7gSDt+EZM6Uyo2gbOZbNwWFCCuLqhLBEWak/iKhwX61M9JeT8NGDdowPie3UdjeXrGRtYd+Isba5fnv90bUKG4M62Z1JVq086K+rtr6u+zu1NbGXGZBEZwQKF002czWEMqzVhG8VANDOUsDRg3aMD4puQUw+QVvzPqx50EBRTipVvq0rNJeK7+6BpjOJuQfKFVkX4ZkLRLg6QOgl9qpdoyrrGL1FZG6vWMwqNoiAaG8i0aMG7QgPFt+46f5ekZm1i97wTX1yrH690bcFnJMODvlWovPtci46VBTsTaVWsT8mCl2tJFgymSy5VqlfJ2GjBu0IDxfSkphk9W7WfEvB0EFBKqli58oZvKV1eqVcrb6YZjqkAoVEi4v0U12lxVnjd/3ElcfBL1Livu0yvVKuUvNGCUX6hapjDv9b7G6TKUUmno+gtKKaU8QgNGKaWUR2jAKKWU8ggNGKWUUh6hAaOUUsojNGCUUkp5hAaMUkopj9CAUUop5REFeqkYETkG7He6jjTKAsedLuISvL0+8P4avb0+8P4avb0+8P8aLzfGlMvqoAIdMN5GRKLcWd/HKd5eH3h/jd5eH3h/jd5eH2iNqbSLTCmllEdowCillPIIDRjvMsHpArLg7fWB99fo7fWB99fo7fWB1gjoGIxSSikP0RaMUkopj9CAyWci0kFEdorIbhF5NpNj7hCRbSKyVUQ+97YaRaSqiCwSkfUisklEOuVzfR+JyFER2ZLJ90VERrvq3yQijb2svrtddW0SkV9E5Or8rM+dGtMc11REkkWkR37Vlua5s6xRRG4QkQ2u98oSb6pPREqIyGwR2eiqr28+11fF9T7d7nr+xzM4xrPvFWOMfuXTFxAA7AGuAIKBjUDddMfUBNYDpVzXy3thjROAh12X6wL78rnG1kBjYEsm3+8EzAMEiAR+9bL6WqT5/+2Y3/W5U2Oa34WfgblAD2+rESgJbAOquq7n93slq/qeB0a6LpcDTgDB+VhfJaCx63Ix4LcM3ssefa9oCyZ/NQN2G2P2GmMSgOlA13THDADGGmNOAhhjjnphjQYo7rpcAjiUj/VhjFmKfbNmpivwsbFWASVFpFL+VJd1fcaYX1L/f4FVQHi+FHZxDVn9DAEeBb4G8vt3EHCrxruAb4wxB1zH52udbtRngGIiIkBR17FJ+VEbgDHmsDFmnevyGWA7UDndYR59r2jA5K/KwME016P55394LaCWiKwQkVUi0iHfqrPcqXEYcI+IRGM/3T6aP6W5zZ3X4C36YT9BehURqQzcBoxzupZLqAWUEpHFIrJWRO5zuqB0xgB1sB/ANgOPG2NSnChERKoB1wC/pvuWR98rgXn1QMotksFt6afxBWK7yW7AfrJdJiL1jTF/ebi2VO7U2BuYYoz5n4g0Bz5x1ejImycD7rwGx4lIG2zAtHK6lgy8AzxjjEm2H8C9UiDQBLgRCANWisgqY8xvzpZ1wc3ABqAtcCWwQESWGWNO52cRIlIU2xIdksFze/S9ogGTv6KBKmmuh/PP7qVoYJUxJhH4XUR2YgNnTf6U6FaN/YAOAMaYlSISil3XyJGulAy48xocJSINgQ+BjsaYGKfryUAEMN0VLmWBTiKSZIz5ztmyLhINHDfGnAXOishS4GrsWIM36AuMMHawY7eI/A7UBlbnVwEiEoQNl8+MMd9kcIhH3yvaRZa/1gA1RaS6iAQDvYBZ6Y75DmgDICJlsd0Ae72sxgPYT42ISB0gFDiWjzVmZRZwn2uGTCRwyhhz2OmiUolIVeAb4F4v+rR9EWNMdWNMNWNMNWAG8IiXhQvATOA6EQkUkcLAtdhxBm+R9n1SAbiKfHwvu8Z+JgHbjTFvZXKYR98r2oLJR8aYJBEZDMzHztD5yBizVUSGA1HGmFmu77UXkW1AMvBUfn7CdbPGfwETRWQotjndx/UpLV+IyDRsF2JZ1zjQv4EgV/3jsONCnYDdQBz2k2S+caO+l4EywPuuFkKSyeeFEd2o0XFZ1WiM2S4iPwCbgBTgQ2PMJadd52d9wCvAFBHZjO2KesYYk58rLLcE7gU2i8gG123PA1XT1OjR94qeya+UUsojtItMKaWUR2jAKKWU8ggNGKWUUh6hAaOUUsojNGCUUkp5hAaMUkopj9CAUcoHuJal/z63xyiVnzRglMoDrjOh9f2kVBr6hlAqh0Skmmszp/eBdcC9IrJSRNaJyFeuRQYRkU4iskNElrs2d8q0lSEizcRuQrbe9e9VGRwzTEQ+EZGfRWSXiAxI8+2iIjLD9XyfuZYLQUReFpE1IrJFRCak3q6UJ2nAKJU7VwEfAzdhFwFtZ4xpDEQBT7gWAh2PXdSyFXbjqUvZAbQ2xlyDXVLmv5kc1xDoDDQHXhaRy1y3XwMMwW4EdwV2uRCAMcaYpsaY+tiVh2/J9itVKpt0LTKlcme/MWaViNyC/aO+wtU4CAZWYlfP3WuM+d11/DRg4CUerwQwVURqYtd5C8rkuJnGmHPAORFZhN0o7i9gtTEmGsC1/lQ1YDnQRkSeBgoDpYGtwOycvWSl3KMBo1TunHX9K8ACY0zvtN8UkWuy+XivAIuMMbe5NolanMlx6RcRTL0en+a2ZCDQ1Yp6H4gwxhwUkWHYFbCV8ijtIlMqb6wCWopIDQARKSwitbBdXle4wgLgziwepwTwh+tyn0sc11VEQkWkDHZF30vtF5QaJsdd40I9sqhBqTyhAaNUHjDGHMMGwjQR2YQNnNqubqxHgB9EZDlwBDh1iYd6A3hdRFZgt0vIzGpgjut5XjHGZLpJlGs31InYbXu/I/82r1MFnC7Xr5SHiUhRY0ysa+bWWGCXMebtXDzeMCDWGDMqr2pUyhO0BaOU5w1wDbhvxXaBjXe4HqXyhbZglHKAiPQFHk938wpjzCAn6lHKEzRglFJKeYR2kSmllPIIDRillFIeoQGjlFLKIzRglFJKeYQGjFJKKY/4P5ahrf/wpf2DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2a8856cf1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (GSearch_6.best_score_, GSearch_6.best_params_))\n",
    "test_means = GSearch_6.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = GSearch_6.cv_results_[ 'std_test_score' ]\n",
    "train_means = GSearch_6.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = GSearch_6.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(GSearch_6.cv_results_).to_csv('my_preds_reg_alpha_reg_lambda_6.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "train_scores = np.array(train_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "\n",
    "for i, value in enumerate(reg_alpha):\n",
    "    pyplot.plot(reg_lambda, -test_scores[i], label= 'reg_alpha:'   + str(value))\n",
    "\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'reg_alpha' )                                                                                                      \n",
    "pyplot.ylabel( '-Log Loss' )\n",
    "pyplot.savefig( 'reg_alpha_vs_reg_lambda6.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到此最后的reg_alpha，reg_lambda的参数也已确定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画曲线的目的是如果，当曲线变化的时候，他对模型性能的一个影响，如果这一步没有找到最佳参数，我们需要调整的时候，应该是往大里调还是小里调，"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
